Head Wearable Device, System, and Method for Displaying Teamed Asset Information

ABSTRACT

A head wearable device, a method, and a system. The head wearable device may include a display, a head tracking system, a user input system, and a processor. The processor may be configured to output a stream of image data to the display for presentation to the user, the image data associated with images aligned with a determined position and a determined orientation of the head of the user relative to an environment, the images including a user-selectable depiction of a teamed asset. The processor may be further configured to receive user input data from the user input system, wherein the user input data includes user selection data associated with a selected teamed asset. The processor may be further configured to update the stream of image data associated with the images such that the images further include a depiction of information associated with the selected teamed asset.

BACKGROUND

The next generation of warfighters will operate in highly complex anddynamic battlespace, which increase the demands placed on humanoperators and/or pilots. One factor contributing to the complexity offuture battlespace is manned unmanned teaming (MUM-T) operations. MUM-Toperations describe a scenario in which a manned operator (e.g., in anairborne platform or on the ground) is controlling one or more unmannedplatforms (e.g., unmanned vehicles).

Traditional avionics interfaces and interaction control methods are notsatisfactory for managing and facilitating MUM-T operations. Pilots arecurrently unable to effectively control own-ship and multiple autonomousunmanned aerial system (UAS) assets within a battlespace.

Aviation operators (e.g., ground operators and airborne pilots) arecurrently task saturated due to the high demands of the operators'roles. Operators must manage multiple sensor information feeds andvehicle interfaces to perform mission responsibilities. MUM-T willrequire operators to assume new roles in addition to performing existingtasks. Currently, operators lack intuitive interfaces, which wouldenable the operators to manage the new responsibilities without asignificant increase in workload.

Many of the existing pilot vehicle interfaces require pilots to performhead down data entry for extended periods of time. Such head down dataentry redirects the pilots' focus from looking out and managing thebattlespace environment to focusing inside the cockpit. Operatorscurrently lack methods to enhance situational awareness and manageteamed assets in MUM-T operations.

Existing head wearable devices do not offer intuitive interactionmethods to engage with any virtual content visualized on the device.Existing head wearable devices lack interaction and control methods toselect, manipulate, and provide inputs to the computer generated contenton the head wearable device.

SUMMARY

In one aspect, embodiments of the inventive concepts disclosed hereinare directed to a head wearable device. The head wearable device mayinclude a display, a head tracking system, a user input system, and aprocessor communicatively coupled to the display, the head trackingsystem, and the user input system. The display may be implemented in oron the head wearable device and configured to present imagery to a userof the head wearable device. The head tracking system may be implementedin or on the head wearable device. The head tracking system may beconfigured to determine a position and an orientation of a head of theuser of the head wearable device relative to an environment, and outputposition and orientation data associated with the determined positionand the determined orientation of the head of the user relative to theenvironment. The user input system may include at least one of an eyetracking system or a voice recognition system. The user input system maybe configured to detect user inputs of the user of the head wearabledevice and output user input data. The processor may be configured tooutput a stream of image data to the display for presentation to theuser, the image data associated with images aligned with the determinedposition and the determined orientation of the head of the user relativeto the environment, the images including a user-selectable depiction ofa teamed asset. The processor may be further configured to receive theuser input data from the user input system, wherein the user input dataincludes user selection data associated with a selected teamed asset.The processor may be further configured to update the stream of imagedata associated with the images such that the images further include adepiction of information associated with the selected teamed asset.

In a further aspect, embodiments of the inventive concepts disclosedherein are directed to a method. The method may include outputting, by aprocessor of a head wearable device, a stream of image data to a displayof the head wearable device for presentation to a user of the headwearable device, the image data associated with images aligned with adetermined position and a determined orientation of a head of the userrelative to an environment, the images including a user-selectabledepiction of a teamed asset. The method may further include receiving,by the processor of the head wearable device, user input data from auser input system, wherein the user input data includes user selectiondata associated with a selected teamed asset. The method may furtherinclude updating, by the processor of the head wearable device, thestream of image data associated with the images such that the imagesfurther include a depiction of information associated with the selectedteamed asset.

In a further aspect, embodiments of the inventive concepts disclosedherein are directed to a system. The system may include a computingdevice and a head wearable device. The computing device may include aprocessor and may be configured to output data. The head wearable devicemay be communicatively coupled to the computing device. The headwearable device may include a display, a head tracking system, a userinput system, and a processor communicatively coupled to the display,the head tracking system, and the user input system. The display may beimplemented in or on the head wearable device and configured to presentimagery to a user of the head wearable device. The head tracking systemmay be implemented in or on the head wearable device. The head trackingsystem may be configured to determine a position and an orientation of ahead of the user of the head wearable device relative to an environment,and output position and orientation data associated with the determinedposition and the determined orientation of the head of the user relativeto the environment. The user input system may include at least one of aneye tracking system or a voice recognition system. The user input systemmay be configured to detect user inputs of the user of the head wearabledevice and output user input data. The processor may be configured tooutput a stream of image data to the display for presentation to theuser, the image data associated with images aligned with the determinedposition and the determined orientation of the head of the user relativeto the environment, the images including a user-selectable depiction ofa teamed asset. The processor may be further configured to receive theuser input data from the user input system, wherein the user input dataincludes user selection data associated with a selected teamed asset.The processor may be further configured to update the stream of imagedata associated with the images such that the images further include adepiction of information associated with the selected teamed asset.

BRIEF DESCRIPTION OF THE DRAWINGS

Implementations of the inventive concepts disclosed herein may be betterunderstood when consideration is given to the following detaileddescription thereof. Such description makes reference to the includeddrawings, which are not necessarily to scale, and in which some featuresmay be exaggerated and some features may be omitted or may berepresented schematically in the interest of clarity. Like referencenumerals in the drawings may represent and refer to the same or similarelement, feature, or function. In the drawings:

FIG. 1 is a view of an exemplary embodiment of a system according to theinventive concepts disclosed herein.

FIG. 2 is a view of the input/output devices of the aircraft of FIG. 1according to the inventive concepts disclosed herein.

FIG. 3 is a view of the aircraft sensors of the aircraft of FIG. 1according to the inventive concepts disclosed herein.

FIG. 4 is a view of the head wearable device of the aircraft of FIG. 1according to the inventive concepts disclosed herein.

FIG. 5 is a view of the eye tracking system of the head wearable deviceof FIG. 4 according to the inventive concepts disclosed herein.

FIG. 6 is a view of the head tracking system of the head wearable deviceof FIG. 4 according to the inventive concepts disclosed herein.

FIG. 7 is a view of the voice recognition system of the head wearabledevice of FIG. 4 according to the inventive concepts disclosed herein.

FIG. 8 is a view of exemplary devices of the aircraft of FIG. 1communicatively coupled via a data network switch according to theinventive concepts disclosed herein.

FIG. 9 is a view of a pilot wearing the head wearable device in acockpit of the aircraft of FIG. 1 according to the inventive conceptsdisclosed herein.

FIG. 10 is a view of a computing device having an integrated HDD of thecontrol station 126 of FIG. 1 according to the inventive conceptsdisclosed herein.

FIG. 11 is a view of a view of an exemplary image displayed by thedisplay of the head wearable device of FIG. 1 according to the inventiveconcepts disclosed herein.

FIG. 12 is a view of a view of an exemplary image displayed by thedisplay of the head wearable device of FIG. 1 according to the inventiveconcepts disclosed herein.

FIG. 13 is a view of a view of an exemplary image displayed by thedisplay of the head wearable device of FIG. 1 according to the inventiveconcepts disclosed herein.

FIG. 14 is a view of a view of an exemplary image displayed by thedisplay of the head wearable device of FIG. 1 according to the inventiveconcepts disclosed herein.

FIG. 15 is a view of a view of an exemplary image displayed by thedisplay of the head wearable device of FIG. 1 according to the inventiveconcepts disclosed herein.

FIG. 16 is a view of a view of an exemplary image displayed by thedisplay of the head wearable device of FIG. 1 according to the inventiveconcepts disclosed herein.

FIG. 17 is a view of a view of an exemplary image displayed by thedisplay of the head wearable device of FIG. 1 according to the inventiveconcepts disclosed herein.

FIG. 18 is a view of a view of an exemplary image displayed by thedisplay of the head wearable device of FIG. 1 according to the inventiveconcepts disclosed herein.

FIG. 19 is a diagram of an exemplary embodiment of a method according tothe inventive concepts disclosed herein.

DETAILED DESCRIPTION

Before explaining at least one embodiment of the inventive conceptsdisclosed herein in detail, it is to be understood that the inventiveconcepts are not limited in their application to the details ofconstruction and the arrangement of the components or steps ormethodologies set forth in the following description or illustrated inthe drawings. In the following detailed description of embodiments ofthe instant inventive concepts, numerous specific details are set forthin order to provide a more thorough understanding of the inventiveconcepts. However, it will be apparent to one of ordinary skill in theart having the benefit of the instant disclosure that the inventiveconcepts disclosed herein may be practiced without these specificdetails. In other instances, well-known features may not be described indetail to avoid unnecessarily complicating the instant disclosure. Theinventive concepts disclosed herein are capable of other embodiments orof being practiced or carried out in various ways. Also, it is to beunderstood that the phraseology and terminology employed herein is forthe purpose of description and should not be regarded as limiting.

As used herein a letter following a reference numeral is intended toreference an embodiment of the feature or element that may be similar,but not necessarily identical, to a previously described element orfeature bearing the same reference numeral (e.g., 1, 1 a, 1 b). Suchshorthand notations are used for purposes of convenience only, andshould not be construed to limit the inventive concepts disclosed hereinin any way unless expressly stated to the contrary.

Further, unless expressly stated to the contrary, “or” refers to aninclusive or and not to an exclusive or. For example, a condition A or Bis satisfied by anyone of the following: A is true (or present) and B isfalse (or not present), A is false (or not present) and B is true (orpresent), and both A and B are true (or present).

In addition, use of the “a” or “an” are employed to describe elementsand components of embodiments of the instant inventive concepts. This isdone merely for convenience and to give a general sense of the inventiveconcepts, and “a” and “an” are intended to include one or at least oneand the singular also includes the plural unless it is obvious that itis meant otherwise.

Finally, as used herein any reference to “one embodiment,” or “someembodiments” means that a particular element, feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the inventive concepts disclosed herein.The appearances of the phrase “in some embodiments” in various places inthe specification are not necessarily all referring to the sameembodiment, and embodiments of the inventive concepts disclosed mayinclude one or more of the features expressly described or inherentlypresent herein, or any combination of sub-combination of two or moresuch features, along with any other features which may not necessarilybe expressly described or inherently present in the instant disclosure.

Broadly, embodiments of the inventive concepts disclosed herein aredirected to a method, system, and head wearable device including aprocessor, a display, a head tracking system, and a user input system(e.g., an eye tracking system and/or a voice recognition system). Thehead wearable device may provide a user (e.g., an operator on the groundor in the air) with enhanced battlespace management capabilities and anintuitive interface to command and manage MUM-T operations. Theprocessor may be configured to generate graphical images (e.g.,symbology, text, and/or depictions), which, for example, may representteamed assets and battlespace features. Such generated graphical imagesmay be aligned with a determined position and a determined orientationof the head of the user relative to an environment or environmentalfeature and output to the display for presentation to the user. Assetsmay refer to vehicles (manned vehicles and/or unmanned vehicles; e.g.,aircraft (e.g., manned aircraft and/or unmanned aerial systems (UASs)),spacecraft, ground vehicles (e.g., automobiles and/or tanks), ships,and/or submarines), soldiers, military installations, and/or bases;likewise, teamed assets may refer to friendly assets. For example,battlespace features may include sensor ranges, weapon ranges, targetinginformation, landing zones, and/or threat areas. Additionally, theintegration of the eye tracking system and/or the voice recognitionsystem with the head wearable device allows for new user interfacecapabilities, such as for managing teamed assets in a battlespace. Thehead wearable device may be implemented as a virtual reality headwearable device and/or an augmented reality head wearable device. Forexample, the head wearable device may provide pilots and/or groundoperators a means for enhanced battlespace management and an intuitiveinterface to command and manage MUM-T operations. The head wearabledevice may assist operators in managing complex threat environments andintensive joint force operations. The head wearable device may enable amanned operator to manage MUM-T operations without a significantincrease in workload. The head wearable device may enable a head-up,eyes-out means for enhanced battlespace situational awareness and mayprovide a solution for operators to direct, visualize, and manage MUM-Toperations.

Some embodiments may include the use of augmented reality or virtualreality symbology to indicate the location of friendly, enemy, neutral,or unknown battlespace entities. For example, the processor of the headwearable device may generate symbology and output the symbology to thedisplay, where the symbology is displayed and overlaid on a location ofreal entities in a battlespace environment. Additional stereoscopic andperspective symbology may be rendered on the display of the headwearable device to illustrate entities' sensor and weapons ranges,targeting information, landing zones, threat areas, and otherbattlespace features. Further, virtual content such as MUM-T menusrelated to asset status, tasks, and capabilities may be generated by theprocessor and displayed by the display based on mission context. Someembodiments may include a human machine interface (HMI) and alertingschema to direct a pilot's view toward threats and other assets, whichare out of the pilot's field of view. Such alerting schema may alert thepilot of a high priority threat or high priority information thatrequires immediate action. The HMI and alerting schema may direct thepilot toward a direction of the asset and/or threat or display criticalinformation within the pilot's current field of view. Additionally, thehead wearable device may be configured to display sensor and targetinginformation transmitted from other teamed assets (e.g., from teamedUASs). Further, the head wearable device and a head down display (HDD)may be communicatively coupled and synched such that the pilot, by usingthe head wearable device and/or the HDD, can select a teamed asset and aformat will be displayed adjacent to that asset, for example, to presentsensor feeds, targeting information, streaming video, images, voice, orother data from the selected teamed asset. In some embodiments, thepilot may select a threat area or a particular enemy asset and a formatwill be displayed adjacent to the selected threat area or the particularenemy asset to show intelligence information being collected on theselected threat area or the particular enemy asset, where suchintelligence information may be transmitted and provided from any ofvarious teamed assets.

In some embodiments, the head wearable device may be used to showclassified content to a wearing user of the head wearable device in anunclassified environment (e.g., a room with people not authorized toview classified information or in a cockpit/aircraft with unclassifiedcrew members) such that only the wearing user may view classifiedcontent on the head wearable device while other people, who might lack arequisite classified clearance level, near the wearing user might onlysee unclassified or less classified information on other displays in theopen environment. For example, some MUM-T operations and datalinks willinvolve classified information; however, not all platforms involved inMUM-T operations may be authorized to handle classified information. Assuch, the head wearable device may be implemented as a secured devicecertified to store and display classified information such that only thecleared operators wearing the device would see the classified contentand could bring the head wearable device to other platforms to work inunclassified environments. The head wearable device may be used to showclassified content in the in the battlespace, overlay virtual content ona HDD, or show virtual displays and/or formats of classified content.For example, the head wearable device may limit the need for entireplatforms to be classified or for an entire flight crew to be cleared tohandle classified content.

Referring now to FIG. 1, an exemplary embodiment of a system 100according to the inventive concepts disclosed herein includes at leastone aircraft 102, a control station 126, satellites 132, globalpositioning system (GPS) satellites 134, a network 136, a networkoperations center (NOC) 138, and UASs 142. Some or all of the aircraft102, the control station 126, the satellites 132, the GPS satellites134, the network 136, the NOC 138, and the UASs 142 may becommunicatively coupled at any given time.

The aircraft 102 includes at least one communication system 104, aplurality of computing devices 112 (which may also be referred to asaircraft computing devices, helicopter computing devices, or vehicularcomputing devices as may be appropriate), a GPS device 120, aircraftsensors 122, input/output devices 124, and at least one head wearabledevice 140, as well as other systems, equipment, and devices commonlyincluded in aircraft. Some or all of the communication system 104, thecomputing devices 112, the GPS device 120, the aircraft sensors 122, theinput/output devices 124, the head wearable device 140, and any othersystems, equipment, and devices commonly included in the aircraft 102may be communicatively coupled. While not shown, in some embodiments,the aircraft 102 may optionally include a NOC or include components(e.g., at least one computing device 112 and/or the communication system104) configured to perform functionality similar to the NOC 138. Theaircraft 102 may be implemented as any suitable aircraft, such as ahelicopter or airplane. The aircraft 102 may be a manned aircraft. Whilethe system 100 is exemplarily shown as including the aircraft 102, insome embodiments the inventive concepts disclosed herein may beimplemented in or on any suitable vehicle (e.g., an automobile, train,submersible craft, watercraft, or spacecraft) or in any suitableenvironment.

The communication system 104 includes one or more antennas 106 (e.g.,two antennas 106, as shown), a processor 108, and memory 110, which arecommunicatively coupled. The communication system 104 (such as via oneor more of the antennas 106) is configured to send and/or receivesignals, data, messages, and/or voice transmissions to and/or from thecontrol station 126, other vehicles (e.g., the UASs 142), the satellites132, the NOC 138, and combinations thereof, as well as any othersuitable devices, equipment, or systems. That is, the communicationsystem 104 is configured to exchange (e.g., bi-directionally exchange)signals, data, messages, and/or voice communications with any othersuitable communication system (e.g., which may be implemented similarlyand function similarly to the communication system 104). Additionally,for example, the communication system 104 may be configured to exchange,send, and/or receive (e.g., via a wireless connection, a cabledconnection, and/or a wired connection, or a combination thereof)signals, data, messages, and/or voice communications with, to, and/orfrom any suitable onboard device(s).

The communication system 104 may include at least one processor 108configured to run or execute various software applications, computercode, and/or instructions stored (e.g., maintained) in at least onenon-transitory computer-readable medium (e.g., at least onecomputer-readable medium implemented as hardware; e.g., at least onenon-transitory processor-readable medium, at least one memory 110 (e.g.,at least one nonvolatile memory, at least one volatile memory, or acombination thereof; e.g., at least one random-access memory, at leastone flash memory, at least one read-only memory (ROM) (e.g., at leastone electrically erasable programmable ROM (EEPROM)), at least oneon-processor memory (e.g., at least one on-processor cache, at least oneon-processor buffer, at least one on-processor flash memory, at leastone on-processor EEPROM, or a combination thereof), or a combinationthereof), at least one storage device (e.g., at least one hard-diskdrive, at least one tape drive, at least one solid-state drive, at leastone flash drive, at least one readable and/or writable disk of at leastone optical drive configured to read from and/or write to the at leastone readable and/or writable disk, or a combination thereof), or acombination thereof). Some or all of the at least one computer-readablemedium may be communicatively coupled. For example, the processor 108may be configured to receive data from the computing devices 112 and/orthe head wearable device 140 and execute instructions configured tocause a particular antenna of the antennas 106 to transmit the data as asignal(s) to the UASs 142 and/or another communication system (e.g.,128) of the system 100. Likewise, for example, the processor 108 may beconfigured to route data received as a signal(s) by a particular antennaof the antennas 106 to one or more of the computing devices 112 and thehead wearable device 140. In some embodiments, the processor 108 may beimplemented as one or more radiofrequency (RF) processors.

Each of the antennas 106 may be implemented as or may include anysuitable antenna or antenna device. For example, the antennas 106 may beimplemented as or include at least one electronically scanned array(ESA) (e.g., at least one active ESA (AESA)), at least one radio (e.g.,at least one software defined radio (SDR)), at least one transmitter, atleast one receiver, at least one transceiver, or a combination thereof.

While the communication system 104 is shown as having two antennas 106,one processor 108, and memory 110, the communication system 104 mayinclude any suitable number of antennas 106, processors 108, and memory110. Further, the communication system 104 may include other components,such as a storage device (e.g., solid state drive or hard disk drive),radio tuners, and controllers.

Each of the computing devices 112 of the aircraft 102 may include atleast one processor 114, memory 116, and storage 118, as well as othercomponents, equipment, and/or devices commonly included in a computingdevice, all of which may be communicatively coupled to one another. Eachof the computing devices 112 may be configured to route data to eachother as well as to the head wearable device 140 and/or thecommunication system 104 for transmission to an off-board destination(e.g., satellites 132, NOC 138, control station 126, and/or UASs 142).Likewise, each computing device 112 may be configured to receive datafrom another computing device 112 as well as from the head wearabledevice 140 and/or the communication system 104 transmitted fromoff-board sources (e.g., satellites 132, NOC 138, control station 126).The computing device 112 may include or may be implemented as and/or beconfigured to perform the functionality of any suitable aircraft system,such as an engine indication and crew alerting system (EICAS) computingdevice (e.g., 112-2), a flight management system (FMS) computing device(e.g., 112-3), an integrated flight information system (IFIS) computingdevice (e.g., 112-4), an information management system (IMS) computingdevice (e.g., 112-5), an onboard maintenance system (OMS) computingdevice (e.g., 112-6), and a terrain awareness and warning system (TAWS)computing device (e.g., 112-7). (See, e.g., FIG. 8.) The processor 114may be configured to run various software applications or computer codestored (e.g., maintained) in a non-transitory computer-readable medium(e.g., memory 116 or storage 118) and configured to execute variousinstructions or operations. Additionally, for example, the computingdevices 112 or the processors 114 may be implemented as special purposecomputers or special purpose processors configured (e.g., programmed) toexecute instructions for performing any or all of the operationsdisclosed throughout. In some embodiments, the aircraft 102 may includeany suitable number of computing devices 112.

In some embodiments, one of the computing devices 112 may be configuredto receive position and orientation data (e.g., a stream of position andorientation data) from the head wearable device 140, wherein theposition and orientation data is indicative of a position andorientation of the user's head. For example, the computing device 112may be configured to receive position and orientation data from the headtracking system 408 of the head wearable device 140. Based at least onthe position and orientation data, the processor 114 may be configuredto generate a virtual image (e.g., a virtual scene image) correspondingto a field of view of the wearer of the head wearable device 102. Theprocessor may be configured to output the virtual image as virtual imagedata to the head wearable device 140. The processor 114 may beconfigured to generate a stream of virtual images and output the streamof virtual images as a stream of virtual image data to the head wearabledevice 140.

The GPS device 120 receives location data from the GPS satellites 134and may provide vehicular location data (e.g., aircraft location data)to any of various equipment/systems of the aircraft 102 (e.g., thecommunication system 104, the computing devices 112, the aircraftsensors 122, the input/output devices 124, and the head wearable device140). The GPS device 120 may include a GPS receiver and a processor. Forexample, the GPS device 120 may receive or calculate location data froma sufficient number (e.g., at least four) of GPS satellites 134 in viewof the aircraft 102 such that a GPS solution may be calculated. In someembodiments, the GPS device 120 may be implemented as or as part of acomputing device 112, the communication system 104, navigation sensorsof the aircraft sensors 122, and/or one of the input/output devices 124.The GPS device 120 may be configured to provide the location data to anyof various equipment/systems of a vehicle. For example, the GPS device120 may provide location data to the computing devices 112, thecommunication system 104, head wearable device 140, and the input/outputdevices 124. Further, while FIG. 1 depicts the GPS device 120implemented in the aircraft 102, in other embodiments, the GPS device120 may be implemented in or on any type of vehicle, such asautomobiles, spacecraft, trains, watercraft, or submersible craft.

While the communication system 104, the computing devices 112, the GPSdevice 120, the aircraft sensors 122, the input/output devices 124, andthe head wearable device 140 of the aircraft 102 have been exemplarilydepicted as being implemented as separate devices or systems, in someembodiments, some or all of the communication system 104, the computingdevices 112, the GPS device 120, the aircraft sensors 122, and/or theinput/output devices 124 may be implemented as a single integratedsystem or device or as any number of integrated and/or partiallyintegrated systems and/or devices.

The control station 126 may include at least one communication system128, at least one computing device 130, at least one head wearabledevice 144, and input/output devices 146, as well as other systems,equipment, and devices commonly included in a control station. Some orall of the communication system 128, the computing device 130, the headwearable device 144, the input/output devices 146, and other systems,equipment, and devices commonly included in a control station may becommunicatively coupled. The control station 126 may be implemented as afixed location ground control station (e.g., a ground control station ofan air traffic control tower, or a ground control station of a networkoperations center (e.g., 138)) located on the ground of the earth. Insome embodiments, the control station 126 may be implemented as a mobileground control station (e.g., a ground control station implemented on anon-airborne vehicle (e.g., an automobile or a ship) or a trailer). Insome embodiments, the control station 126 may be implemented as an aircontrol station implemented on an airborne vehicle (e.g., aircraft). Thecontrol station 126 may include a NOC or be communicatively coupled tothe NOC 138 (e.g., via the network 136). For example, an operatorlocated at the control station 126 may wear the head wearable device 144to and be able to direct, visualize, and control MUM-T operations.

The communication system 128 and components thereof (such as antenna106) of the control station 126 may be implemented similarly to thecommunication system 104 except that, in some embodiments, thecommunication system 128 may be configured for operation at a fixedlocation. The computing device 130 and components thereof (such as aprocessor (not shown) and memory (not shown)) of the control station 126may be implemented similarly to the computing devices 112. The headwearable device 144 may be implemented similarly to the head wearabledevice 140 except that, in some embodiments, the head wearable device140 may be configured for operation at a fixed location. Theinput/output devices 146 may be implemented similarly to input/outputdevices 124 except that, in some embodiments, the input/output devices146 may be configured for operation at a fixed location.

While the antennas 106 are exemplarily depicted as being implemented inthe aircraft 102 and the control station 126, in some embodiments,antennas 106 may be implemented in, on, or coupled to any other suitabledevice, equipment, or system, such as the UASs 142, a computing device(e.g., a laptop computing device, a mobile computing, a wearablecomputing device, or a smart phone), a mobile communication system(e.g., a man pack communication system), or satellites 132.

The network 136 may be implemented as any suitable network orcombination of networks. For example, the network 136 may include or beimplemented as the internet, a portion of the internet (such as asecured optical fiber network), an intranet, a wide area network (WAN),a local area network (LAN), and/or a mobile telecommunications network(e.g., a third generation (3G) network or a fourth generation (4G)network)). While the system 100 is exemplarily shown as including thenetwork 136, the system 100 or various components of the system 100 mayinclude or be communicatively coupled via any suitable number and anysuitable types of networks.

The NOC 138 may connect a particular type of communications (e.g.,satellite communications with the satellites 132 and/or aircraftcommunications with the aircraft 102 and/or the UASs 142) with thenetwork 136.

The UASs 142 may be implemented similarly to the aircraft 102, exceptthat, the UASs are configured for unmanned operation.

While FIG. 1 exemplarily includes elements as shown, in someembodiments, one or more of the elements of the system 100 may beomitted, or the system 100 may include other elements. For example, oneor more of the GPS satellites 134, satellites 132, the control station126, the network 136, the NOC 138, or the UASs 142 may be optional.Additionally, while an embodiment has been depicted as including onecontrol station (e.g., the control station 126), other embodiments mayinclude any number of control stations of various types positioned ormoving anywhere in the system 100.

Referring now to FIG. 2, the input/output devices 124 of the aircraft102 of FIG. 1 may include one or more displays (e.g., at least onehead-up display (HUD), at least one HDD, at least one adaptive flightdisplay (AFD), or a combination thereof), at least one eye trackingsystem 206, speakers 216, flight controls 218, at least one keyboard220, at least one microphone 222, at least one voice recognition system224, or a combination thereof, some or all of which may becommunicatively coupled at any given time. While FIG. 2 depicts thevarious exemplary input/output devices 124, the input/output devices 124may include any suitable input/output devices.

For example, the displays of the input/output devices 124 may includetwo HUDs 202-1, 202-2 (which may collectively be referred to as HUDs202) and four HDDs 204-1, 204-2, 204-3, 204-4 (which may collectively bereferred to as HDDs 204). Each of the HUDs 202 and the HDDs 204 may beconfigured to present streams of images (e.g., as video or still images)to a user (e.g., a pilot or an operator). In some embodiments, the HUDs202 and/or HDDs 204 may be implemented as or include a touchscreendisplay. In some embodiments, one or more of the HUDs 202 and the HDDs204 may include an integrated computing device (which may be implementedand function similarly to one of the computing devices 112 of FIG. 1)and/or integrated computing device components (which may be implementedand function similarly to components of one of the computing devices 112of FIG. 1). Each of the HUDs 202 and the HDDs 204 may be communicativelycoupled to one or more of the computing devices 112, the communicationsystem 104, the GPS device 120, the head wearable device 140, other ofthe input/output devices 124, and/or the aircraft sensors 122 of FIG. 1.

The eye tracking system 206 is configured to track eye gestures, trackmovement of a user's eye, track a user's gaze, and/or otherwise receiveinputs from a user's eyes. The eye tracking system 206 may be configuredfor performing fully automatic eye tracking operations of users in realtime. The eye tracking system 206 may include at least one sensor 208,at least one processor 210, a memory 212, and a storage 214, as well asother components, equipment, and/or devices commonly included in an eyetracking system. The sensor 208, the processor 210, the memory 212, andthe storage 214, as well as the other components, equipment, and/ordevices commonly included in the eye tracking system 206 may becommunicatively coupled.

Each sensor 208 may be implemented as any of various sensors suitablefor an eye tracking system. For example, the at least one sensor 208 mayinclude or be implemented as one or more optical sensors (e.g., at leastone camera configured to capture images in the visible light spectrumand/or the infrared spectrum). In some embodiments, the at least onesensor 208 is one or more dedicated eye tracking system sensors. Whilethe sensor 208 has been exemplarily depicted as being included in theeye tracking system 206, in some embodiments, the sensor 208 may beimplemented external to the eye tracking system 206. For example, thesensor 208 may be implemented as an optical sensor (e.g., of the opticalsensors 316 of the aircraft sensors 122) located within the aircraft 102and communicatively coupled to the processor 210.

The processor 210 may be configured to process data received from thesensor 208 and output processed data to one or more onboard devices oronboard systems (e.g., the communication system 104, the computingdevices 112, the aircraft sensors 122, other of the input/output devices124, the head wearable device 140, or a combination thereof). Forexample, the processor 210 may be configured to generate eye trackingdata and output the generated eye tracking data to one of the computingdevices 112 and/or the head wearable device 140. The processor 210 ofthe eye tracking system 206 may be configured to run various softwareapplications or computer code stored (e.g., maintained) in anon-transitory computer-readable medium (e.g., memory 212 and/or storage214) and configured to execute various instructions or operations. Theprocessor 210 may be implemented as a special purpose processorconfigured to execute instructions for performing any or all of theoperations disclosed throughout.

The voice recognition system 224 may be implemented similarly to thevoice recognition system 418 shown in and described with respect toFIGS. 4 and 7.

In some embodiments, some or all of the input/output devices 124 mayinclude an integrated computing device (which may be implemented andfunction similarly to one of the computing devices 112 of FIG. 1) and/orintegrated computing device components (which may be implemented andfunction similarly to components of one of the computing devices 112 ofFIG. 1).

Referring now to FIG. 3, the aircraft sensors 122 of FIG. 1 are shown.Each of the aircraft sensors 122 may be configured to sense a particularcondition(s) external to the aircraft 102 or within the aircraft 102 andoutput data associated with particular sensed condition(s) to one ormore onboard devices or onboard systems (e.g., the communication system104, the computing devices 112, the aircraft sensors 122, theinput/output devices 124, the head wearable device 140, or a combinationthereof). For example, the aircraft sensors 122 may include an inertialmeasurement unit 302, a radio altimeter 304, radar (e.g., weather 306,surveillance radar, and/or weapon radar), airspeed sensors 308, flightdynamic sensors 310 (e.g., configured to sense pitch, roll, and/or yaw),air temperature sensors 312, air pressure sensors 314, optical sensors316 (e.g., cameras configured to capture images in the visible lightspectrum and/or the infrared spectrum, such as forward looking infrared(FLIR) cameras and/or short-wavelength infrared (SWIR) cameras), soundranging sensors (not shown), surveillance sensors 318, equipment sensors320 (e.g., electrical system sensors, hydraulic system sensors, bleedair sensors, environmental conditioning sensors, fuel sensors, and/orfire warning/suppression sensors), and/or engine speed sensors 322, someor all of which may be communicatively coupled at any given time.Additionally, the GPS device 120 may be considered as one of theaircraft sensors 122.

For example, at least some of the aircraft sensors 122 may beimplemented as navigation sensors (e.g., the GPS device 120, theinertial measurement unit 302, a radio altimeter 304, weather radar 306,airspeed sensors 308, flight dynamic sensors 310, air temperaturesensors 312, and/or air pressure sensors 314) configured to sense any ofvarious flight conditions or aircraft conditions typically used byaircraft and output navigation data (e.g., aircraft location data,aircraft orientation data, aircraft direction data, aircraft speed data,and/or aircraft acceleration data). For example, various flightconditions or aircraft conditions may include altitude, aircraftlocation (e.g., relative to the earth), aircraft orientation (e.g.,relative to the earth), aircraft speed, aircraft acceleration, aircrafttrajectory, aircraft pitch, aircraft roll, aircraft yaw, airtemperature, and/or air pressure. For example, the GPS device 120 andthe inertial measurement unit 302 may provide aircraft location data andaircraft orientation data, respectively, to a processor (e.g., aprocessor of the GPS device 120, processor 114, processor 114-1,processor 108, processor 210, processor 402, or a combination thereof).

In some embodiments, some or all of the aircraft sensors 122 may includean integrated computing device (which may be implemented and functionsimilarly to one of the computing devices 112 of FIG. 1) and/orintegrated computing device components (which may be implemented andfunction similarly to components of one of the computing devices 112 ofFIG. 1).

Further, while the aircraft sensors 122 are implemented in or on theaircraft 102, some embodiments may include vehicle sensors implementedon any suitable vehicle according to the inventive concepts disclosedherein.

Referring now to FIGS. 4-7, an exemplary embodiment of the head wearabledevice 140 according to the inventive concepts disclosed herein isshown. The head wearable device 140 may be implemented as an augmentedreality head wearable device or a virtual reality head wearable device.The head wearable device 140 may be implemented as an occluded or atransmissive head wearable device. The head wearable device 140 may beconfigured to receive image data (e.g., augmented reality image dataand/or virtual reality image data) from one of the computing devices112. For example, the head wearable device 140 may be configured toprovide a wearer of the head wearable device 140 with an augmentedreality experience by blending and/or overlaying virtual images withand/or on real images (e.g., real scene images) to create combinedthree-dimensional immersive scene images, which are presented to thewearer of the head wearable device 102. Additionally, for example, thehead wearable device 140 may be configured to provide a wearer of thehead wearable device 140 with a virtual reality experience by blendingand/or overlaying virtual images (e.g., symbology, text, and/ordepictions) with and/or on a synthetic environment.

The head wearable device 140 may be configured to present augmentedreality or virtual reality scene images to the wearer of the headwearable device 140. The head wearable device 140 may include at leastone processor 402, at least one memory 404, at least one storage device406, a head tracking system 408, a user input system (e.g., an eyetracking system 408 and/or a voice recognition system 418), at least onedisplay 412, at least one speaker 414, at least one camera 416, and atleast one biometric sensor 420, some or all of which may becommunicatively coupled.

The processor 402 may be implemented as any suitable processor, such asa general purpose processor, an image processor, and/or afield-programmable gate array (FPGA). The processor 402 may beconfigured to receive data (e.g., image data associated with images(e.g., assets and battlespace features) to be overlaid on a user's fieldof view or overlaid on a virtual image of an environment and/or virtualimage data associated with virtual images of an environment) from one ofthe computing devices 112. Such received data may be aligned withdetermined position and determined orientation of the head of the userrelative to the environment. The processor 402 may be configured togenerate graphical images (e.g., symbology, text, and/or depictions),which, for example, may represent teamed assets and battlespacefeatures. Such generated graphical images may be aligned with determinedposition and determined orientation of the head of the user relative tothe environment. The processor 402 may be configured to output thegraphical images to the display 412 for presentation to the user. Insome embodiments, the graphical images are augmented reality graphicalimages that augment a user's real world view on a see-through display ofthe head wearable device 140. In some embodiments, the graphical imagesare overlaid on virtual images to provide an enhanced virtual realityexperience for a user wearing an occluded head wearable device. Forexample, the processor 402 of the head wearable device 140 may generatesymbology and output the symbology to the display 412, where thesymbology is displayed and overlaid on a location of real entities in abattlespace environment. For example, the symbology may indicate thelocation of friendly, enemy, neutral, or unknown battlespace entities.Additionally, the symbology illustrate entities' sensor and weaponsranges, targeting information, landing zones, threat areas, and otherbattlespace features. Further, the processor 402 may generate virtualcontent, such as such as MUM-T menus related to asset status, tasks, andcapabilities, and output such virtual content to the display 412 basedon mission context. In some embodiments, the processor 402 may generatean alerting schema (e.g., a blinking arrow near an edge of the display412) to direct a pilot's view toward threats and other assets, which areout of the pilot's field of view, and output the alerting schema to thedisplay 412. Additionally, for example, the processor 402 may receivesensor and targeting information transmitted from other teamed assets(e.g., from teamed UASs 142), generate graphical images associated withthe sensor and targeting information, and output the graphical images tothe display 412.

For example, the processor 402 may be configured to generate and outputa stream of image data to the display 412 for presentation to the user.The image data may be associated with images aligned with the determinedposition and the determined orientation of the head of the user relativeto the environment, and the images may including any of various assetsand battlespace features. For example, the images may include at leastone user-selectable depiction of at least one unmanned vehicle (e.g.,UAS 142). The processor 402 may be configured to receive user input datafrom the user input system (e.g., the eye tracking system 408 and/or thevoice recognition system 418), wherein the user input data includes userselection data associated with a selected unmanned vehicle of the atleast one unmanned vehicle. Additionally, the processor 402 may beconfigured to update the stream of image data associated with the imagessuch that the images further include a depiction of informationassociated with the selected unmanned vehicle. For example, thedepiction of information associated with the selected unmanned vehiclemay include a depiction of sensor feed data captured by at least onesensor of the selected unmanned vehicle. Additionally, for example, theinformation associated with the selected unmanned vehicle may includeinformation of at least one of status, tasks, or capabilities of theselected unmanned vehicle.

The processor 402 may be configured to receive user input data from theuser input system. For example, the user input data may include usersselection data associated with user selections of various assets orbattlespace features. Additionally, the user input data may include usercommand data associated with various user commands. For example, theuser input data may include user command data associated with a commandto control a selected unmanned vehicle.

In some embodiment, the image data that the processor 402 outputs to thedisplay 412 is battlespace image data associated with images of abattlespace. For example, the images of the battlespace may includedepiction of battlespace features and assets. For example, the imagesassociated with the image data may include user-selectable depictions ofbattlespace objects, and the processor 402 may receive user selectiondata associated with a selected battlespace object of the battlespaceobjects. In response to receiving the user selection data, the processor402 may update the stream of image data associated with the images suchthat the images further include a depiction of information associatedwith the selected battlespace object. Additionally, for example, theprocessor 402 may receive user command data associated with a command tosurveil the selected battlespace object, and the processor 402 mayoutput a command to control a particular unmanned vehicle to surveil theselected battlespace object. Further, for example, the processor 402 mayreceive user command data associated with a command to attack theselected battlespace object, and the processor 402 may output a commandto control a particular unmanned vehicle to attack the selectedbattlespace object. In some embodiments, each user-selectable depictionof a battlespace object includes graphical information indicative ofwhether each battlespace object is classified as friendly, neutral,enemy, or unknown threat. For example, where the selected battlespaceobject is currently classified as unknown threat, the processor 402 mayreceive user command data associated with a command to classify theselected battlespace object, and the processor 402 may output a commandto classify the selected battlespace object as friendly, neutral, orenemy.

In some embodiments, the processor 402 may generate image dataassociated with a depiction of information classified to a predeterminedclassification level. For example, the processor 402 may be configuredto verify an identity of the user of the head wearable device 140 basedat least on biometric information obtained from the biometric sensor 420and determine a predetermined classification level of the user based atleast on the identity. Additionally, the processor 402 may output astream of image data to the display 412 such that the stream of imagedata corresponds to the predetermined classification level of the user.

In some embodiments, the processor 402 may be configured to receive realimages as real image data from the camera 416. The processor 108 may beconfigured to generate graphical images (e.g., symbology, text, and/ordepictions), which, for example, may represent teamed assets andbattlespace features, and overlay the graphical images on the real imagedata to form combined image data, and the processor 402 may beconfigured to output the combined image data to the display 412.

In some embodiments, the at least one processor 402 may be implementedas a plurality of processors, such as at least one general purposeprocessor and at least one image processor. The processor 402 may beconfigured to run various software applications or computer code storedin a non-transitory computer-readable medium (e.g., memory 404 and/orstorage device 406) and configured to execute various instructions oroperations. The processor 402 may be implemented as a special purposeprocessor configured to execute instructions for performing any or allof the operations disclosed throughout.

The head tracking system 408 may be configured to determine and track aposition and an orientation of a user's head relative to an environment.The head tracking system 408 may be configured for performing fullyautomatic head tracking operations in real time. As shown in FIG. 6, thehead tracking system 408 may include sensors 602, a processor 604,memory 606, and storage 608, as well as other components, equipment,and/or devices commonly included in a head tracking system. The sensors602, the processor 604, the memory 606, and the storage 608, as well asthe other components, equipment, and/or devices commonly included in ahead tracking system may be communicatively coupled.

The processor 604 of the head tracking system 410 may be configured toprocess data received from the sensors 602 and output processed data toone of the computing devices 112 and/or the processor 402 for use ingenerating images aligned with the user's field of view. For example,the processor 604 may be configured to determine and track a positionand orientation of a user's head relative to an environment.Additionally, for example, the processor 604 may be configured togenerate position and orientation data associated with such determinedinformation and output the generated position and orientation data toone of the computing devices 112 and/or the processor 402. The processor604 of the head tracking system 114 may be configured to run varioussoftware applications or computer code stored in a non-transitorycomputer-readable medium and configured to execute various instructionsor operations. The processor 604 may be implemented as a special purposeprocessor configured to execute instructions for performing any or allof the operations disclosed throughout.

The eye tracking system 408 may be configured to track eye gestures,track movement of a user's eye, track a user's gaze, track scanpatterns, determine a location of a vergence point (sometimes referredto as a point of regard) of a user's gaze, determine eye locations,track physiological metrics (e.g., blink rate, pupil dilation, time tofirst fixation, fixation length, and/or fixation count), determine anintra-pupillary distance (IPD) between a user's eyes, determine adirection between a determined location of a user's eye and a determinedlocation of a vergence point for each of a user's eyes, and/or otherwisereceive inputs from a user's eyes. The eye tracking system 408 may beconfigured for performing fully automatic eye tracking operations ofusers in real time. The eye tracking system 408 may include at least onesensor 502, at least one processor 504, memory 506, and storage 508, asshown in FIG. 5, as well as other components, equipment, and/or devicescommonly included in an eye tracking system. The sensor 502, theprocessor 504, the memory 506, and the storage 508, as well as the othercomponents, equipment, and/or devices commonly included in an eyetracking system may be communicatively coupled.

The eye tracking system 408 may be used as a cursor control device toselect virtual content displayed on the display 412. For example, theeye tracking system 408 can be used to select teamed assets in thebattlespace, or select items from a virtual menu. The eye trackingsystem 408 may also be used to trace the user's scan patterns, and alertthe user of critical information which has been overlooked.

The processor 504 of the eye tracking system 408 may be configured toprocess data received from the sensor 502 and output processed data tothe processor 402 and/or one of the computing devices 112. For example,the processor 504 may be configured to determine a location of avergence point of a user's gaze, determine eye locations, determine anintra-pupillary distance (IPD) between a user's eyes, and/or determine adirection between a determined location of a user's eye and a determinedlocation of a vergence point for each of a user's eyes. Additionally,for example, the processor 504 may be configured to generate dataassociated with such determined information and output the generateddata to the processor 402 and/or one of the computing devices 112. Theprocessor 504 of the eye tracking system 408 may be configured to runvarious software applications or computer code stored in anon-transitory computer-readable medium and configured to executevarious instructions or operations. The processor 504 may be implementedas a special purpose processor configured to execute instructions forperforming any or all of the operations disclosed throughout.

The display 412 may be configured to receive a stream of images as astream of image data and present the stream of images to a wearing userof the head wearable device 140. The display 412 may be implemented asany suitable display, such as a see-through display (e.g., MicrosoftHololens) or an occluded display (e.g., Oculus Rift). Additionally,while the display 412 is depicted as a single display, the display 412may be implemented as a plurality of displays, such as one display foreach eye of a user.

The speaker 414 may be configured to receive audio content associatedwith an environment and to present the audio content as audio to awearing user of the head wearable device 140. Additionally, while thespeaker 414 is depicted as a single speaker, the speaker 414 may beimplemented as a plurality of speakers, such as one speaker for each earof a user.

The camera 416 may be configured to capture real images (e.g., realscene images) of a wearing user's field of view. The camera 416's fieldof view may align with a wearing user's field of view. The camera 416may be configured to output the real images as real image data (e.g.,real scene image data) to one of the computing devices 112 and/or theprocessor 402. The camera 416 may be configured to output a stream ofreal images as a stream of real image data to one of the computingdevices 112 and/or the processor 402. In some embodiments, the camera416 may be implemented as a plurality of cameras.

The voice recognition system 418 may include at least one microphone702, at least one processor 704, memory 706, and storage 708, as shownin FIG. 7, as well as other components, equipment, and/or devicescommonly included in a voice recognition system. The microphone 702, theprocessor 704, the memory 706, and the storage 708, as well as the othercomponents, equipment, and/or devices commonly included in an eyetracking system may be communicatively coupled. The voice recognitionsystem 418 may be configured to recognize voice commands or audibleinputs of a user. The voice recognition system 418 may allow the user touse verbal commands as an interaction and control method. The voicerecognition system 418 may be configured to detect user commands andoutput user command data, which, for example, may be used to providecommands to direct MUM-T operations, which may be visualized on the headwearable device 140. Additionally, verbal commands may be used tomodify, manipulate, and declutter content displayed by the head wearabledevice 140. The voice recognition system 418 may be integrated with theeye tracking system 408 so context of user inputs can be inferred. Forexample, while looking at a particular UAS 142, the user can say, “Turnto intercept”, and one of the processors (e.g., 402, 602, and/or 704)may determine, based at least one the eye tracking data, that the useris implying that the command is intended for the particular UAS 142without saying the particular UAS 142's call sign. The processor 704 maybe configured to process data received from the microphone 702 andoutput processed data (e.g., text data) to one of the computing devices112 and/or the processor 402. The processor 704 may be configured to runvarious software applications or computer code stored in anon-transitory computer-readable medium and configured to executevarious instructions or operations.

The biometric sensor 420 may be configured to sense biometricinformation associated with the user. For example, the biometric sensormay be implemented as a fingerprint reader or a retina scanner.

While the processor 402, the memory 404, the storage device 406, thehead tracking system 410, the user input system (e.g., the eye trackingsystem 408 and/or the voice recognition system 418), the display 412,the speaker 414, the camera 416, and the biometric sensor 420 of thehead wearable device 140 have been exemplarily depicted as beingimplemented as separate devices or subsystems, in some embodiments, someor all of the processor 402, the memory 404, the storage device 406, thehead tracking system 410, the user input system (e.g., the eye trackingsystem 408 and/or the voice recognition system 418), the display 412,the speaker 414, the camera 416, and the biometric sensor 420 may beimplemented as a single integrated system or device or as any number ofintegrated and/or partially integrated subsystems and/or devices.

While the head wearable device 140 exemplarily includes elements asshown, in some embodiments, one or more of the elements of the headwearable device 140 may be omitted, or the head wearable device 140 mayinclude other elements.

Referring now to FIG. 8, various exemplary devices of the aircraft 102of FIG. 1 communicatively coupled via a data network switch 802 (e.g.,an avionics full-duplex Ethernet (AFDX) switch) are shown. For example,a plurality of computing devices 112 (e.g., avionics computing devices),the input/output devices 124, the communication system 104, vehicularsensors (e.g., the aircraft sensors 122), the GPS device 120, and thehead wearable device 140 may be communicatively coupled via the datanetwork switch 802. Each of the plurality of avionics computing devices(e.g., 112-1, 112-2, 112-3, 112-4, 112-5, 112-6, 112-7), theinput/output devices 124, the communication system 104, vehicularsensors (e.g., the aircraft sensors 122), and the GPS device 120 may beconfigured to exchange (e.g., send and/or receive) avionics data withone another via the data network switch 802. While the plurality ofcomputing devices 112, the input/output devices 124, the communicationsystem 104, the aircraft sensors 122, the GPS device 120, and the headwearable device 140 are exemplarily shown as being communicativelycoupled via the data network switch 802, in some embodiments some or allof the plurality of computing devices 112, the input/output devices 124,the communication system 104, the vehicular sensors (e.g., the aircraftsensors 122), the GPS device 120, and the head wearable device 140 maybe communicatively coupled via any suitable data networks and via anysuitable data networking components (e.g., at least one bus (e.g.,Aeronautical Radio, Incorporated (ARINC) 429 busses), at least one dataconcentrator, at least one switch, at least one router, or a combinationthereof).

The plurality of computing devices 112 may be implemented as and/orinclude a plurality of vetronics computing devices, such as a pluralityof avionics computing devices (e.g., which may be implemented in one ormore integrated modular avionics (IMA) cabinets). The plurality ofavionics computing devices may include a first avionics computing device112-1, a crew alerting system (CAS) computing device (e.g., an engineindication and crew alerting system (EICAS) computing device 112-2), aflight management system (FMS) computing device 112-3, an integratedflight information system (IFIS) computing device 112-4, an informationmanagement system (IMS) computing device 112-5, an onboard maintenancesystem (OMS) computing device 112-6, a terrain awareness and warningsystem (TAWS) computing device 112-7, a secure server router computingdevice (not shown), an automatic dependent surveillance (ADS) computingdevice (not shown), and a traffic collision avoidance system (TCAS)computing device (not shown), as well as other avionics computingdevices commonly implemented in an aircraft. Additionally, theinput/output devices 124, the communication system 104, the aircraftsensors 122, the data network switch 802, and the GPS device 120 may beconsidered to be devices of the plurality of avionics computing devicesand may be implemented similarly as and function similarly as avionicsdevices (e.g., 112-1, 112-2, 112-3, 112-4, 112-5, 112-6, 112-7) asdisclosed throughout. Each of the plurality of avionics computingdevices (e.g., 112-1, 112-2, 112-3, 112-4, 112-5, 112-6, 112-7, 112-8)may include components, which may be implemented and function similarlyas the components of the computing device 112 shown and described withrespect to FIG. 1. As such, each of the plurality of avionics computingdevices may include at least one processor, memory, and storage, whichmay be implemented and function similarly as the processor 114, thememory 116, and the storage 118, respectively, of the computing device112 shown and described with respect to FIG. 1. For example, the firstavionics computing device 112-1 may include a processor 114-1, memory116-1, and storage 118-1, which may be implemented and functionsimilarly as the processor 114, the memory 116, and the storage 118,respectively, of the computing device 112 shown and described withrespect to FIG. 1.

The plurality of avionics computing devices (e.g., 112-1, 112-2, 112-3,112-4, 112-5, 112-6, 112-7) and/or processors thereof (e.g., 114-1) maybe implemented as special purpose computers (e.g., the first avionicscomputing device 112-1, the EICAS computing device 112-2, the FMScomputing device 112-3, the IFIS computing device 112-4, the IMScomputing device 112-5, the OMS computing device 112-6, the TAWScomputing device 112-7) and/or special purpose processors (e.g., theprocessor 114-1 of the first avionics computing device 112-1 programmedto execute instructions for operations as disclosed throughout, aprocessor of the EICAS computing device 112-2 programmed to executeinstructions for performing EICAS operations as disclosed throughout, aprocessor of the FMS computing device 112-3 programmed to executeinstructions for performing FMS operations as disclosed throughout, aprocessor of the IFIS computing device 112-4 programmed to executeinstructions for performing IFIS operations as disclosed throughout, aprocessor of the IMS computing device 112-5 programmed to executeinstructions for performing IMS operations as disclosed throughout, aprocessor of the OMS computing device 112-6 programmed to executeinstructions for performing OMS operations as disclosed throughout, aprocessor of the TAWS computing device 112-7 programmed to executeinstructions for performing TAWS operations as disclosed throughout)configured to execute instructions for performing any or all of theoperations disclosed throughout.

Additionally, in some embodiments, the data network switch 802 may beimplemented similarly as and function similarly to one of the avionicscomputing devices (e.g., 112-1, 112-2, 112-3, 112-4, 112-5, 112-6,112-7) or include components that function similarly to components ofone of the avionics computing devices. For example, the data networkswitch 802 may include an integrated computing device (which may beimplemented and function similarly to one of the computing devices 112(e.g., one of the avionics computing devices (e.g., 112-1, 112-2, 112-3,112-4, 112-5, 112-6, 112-7))) and/or integrated computing devicecomponents (which may be implemented and function similarly tocomponents of one of the computing devices 112 of FIG. 1).

Further, while the plurality of avionics computing devices has beenexemplarily depicted and described with respect to FIG. 8 as includingthe first avionics computing device 112-1, the EICAS computing device112-2, the FMS computing device 112-3, the IFIS computing device 112-4,the IMS computing device 112-5, the OMS computing device 112-6, and theTAWS computing device 112-7, in some embodiments, the plurality ofavionics computing devices may omit one or more of the described anddepicted avionics computing devices, include additional numbers of suchavionics computing devices, and/or include other types of suitableavionics computing devices.

Referring now to FIG. 9, a pilot wearing the head wearable device 140 ina cockpit 900 of the aircraft 102 of FIG. 1 is shown. Content displayedby the head wearable device 140 and content displayed by one or more ofdisplays (e.g., HDD 204-1, HDD 204-2, HDD 204-3 and/or HUD 202-1) of thecockpit 900 may be synched such that the pilot, by using the headwearable device 140 and/or one or more of the displays of the cockpit900, can select a teamed asset and a format will be displayed adjacentto that asset, for example, to present sensor feeds, targetinginformation, streaming video, images, voice, or other data from theselected teamed asset. In some embodiments, the content displayed by thehead wearable device 140 and the content displayed by one or more ofdisplays of the cockpit 900 provide a seamless transition between whatvirtual content the pilot is seeing on the head wearable device 140 andthe HDDs 204-1, 204-2, 204-3. In some embodiments, one of the HDDs 204may display a menu with user-selectable options to filter what contentis displayed on the display 412 of the head wearable device 140. Forexample, the HDD menu may provide user-selectable modes for the pilot todeclutter content such as friendly forces, enemy forces, ground assets,air assets, engagement areas, and/or sensor and weapon ranges, forexample, based on the pilot's preference. Additionally, the headwearable device 140 and the HDDs 204 may offer complementary features sothat the operator can easily switch between head-up and head downoperations. For example, when an object or asset is selected on the headwearable device 140 or one of the HDDs 204, the selected object or assetmay also be highlighted to the pilot on the other display device.Additionally, for example, when the pilot directs a scenario or commandsan asset depicted by the head wearable device 140, a correspondingchange may be displayed on one of the HDDs 204, and vice-a-versa.Further, for example, if the pilot selects an asset's sensor feed,targeting feed, or video feed via the head wearable device 140, one ofthe HDDs 204 may display a magnified or enlarged version of the selectedfeed. Similarly, if the pilot selects a feed via one of the HDDs 204,the head wearable device 140 may show that feed on the head wearabledevice 140.

Referring now to FIG. 10, an operator wearing the head wearable device144 may interface with a computing device 130 having an integrated HDDof the control station 126 of FIG. 1 is shown. Content displayed by thehead wearable device 144 and content displayed by computing device 130with the integrated HDD may be synched, similarly as described withrespect to FIG. 9. For example, the operator, by using the head wearabledevice 144 and/or HDD of the computing device 130, can select a teamedasset and a format will be displayed adjacent to that asset, forexample, to present sensor feeds, targeting information, streamingvideo, images, voice, or other data from the selected teamed asset.

Referring now to FIGS. 11-18, views of various exemplary imagesdisplayed by the display 412 of the head wearable device 140 are shown.

Referring now to FIG. 11, a view of an exemplary image displayed by thedisplay 412 of the head wearable device 140 is shown. The image includestwo user-selectable depictions of two unmanned vehicles (e.g., teamedUASs 142-1, 142-2), an eye tracking cursor 1102, a user-selectabledepiction of an unknown threat 1104, and a user-selectable depiction ofan enemy asset 1106 (e.g., an enemy aircraft).

Referring now to FIG. 12, a view of an exemplary image displayed by thedisplay 412 of the head wearable device 140 is shown. By looking nearthe user-selectable depiction of the unknown threat 1104 and/or byspeaking a voice command, the user input system (e.g., the eye trackingsystem 408 and/or the voice recognition system 418) may output userselection data to the processor 402 indicative of the user selecting theuser-selectable depiction of the unknown threat 1104. In response, theprocessor 402 may update the stream of image data to include graphicalcontent of an actions menu 1202 and output the updated stream of imagedata to the display 412. The actions menu 1202 may include one or moreuser-selectable options (e.g., actions), such as to send a UAS, toperform sound ranging, to perform surveillance radar, to perform weaponlocation radar, and/or to send ground assets.

Referring now to FIG. 13, a view of an exemplary image displayed by thedisplay 412 of the head wearable device 140 is shown. By utilizing theuser input system (e.g., the eye tracking system 408 and/or the voicerecognition system 418), the user of the head wearable device 140 mayselect an item from a list (e.g., the actions menu 1202). For example,by utilizing the eye tracking system 408, the user of the head wearabledevice 140 may select an item from a list (e.g., the actions menu 1202)by looking at the item, and the user may confirm the selection by usinga second user input, such as a voice command (e.g., via the voicerecognition system 418) or by pressing a button. For example, byutilizing the voice recognition system 418, the user of the headwearable device 140 may select an item from a list (e.g., the actionsmenu 1202) by speaking a voice command to select the item, andoptionally, the user may confirm the selection by using a second userinput, such as a another voice command (e.g., via the voice recognitionsystem 418) or by pressing a button. Additionally, for example, bylooking near the user-selectable option of “UAS” and/or by speaking avoice command, the user input system (e.g., the eye tracking system 408and/or the voice recognition system 418) may output user selection datato the processor 402 indicative of the user selecting theuser-selectable option to activate a “UAS”. In response, the processor402 may update the stream of image data to include graphical content ofan “ACTIVATE UAS” confirmation 1302 and output the updated stream ofimage data to the display 412.

Referring now to FIG. 14, a view of an exemplary image displayed by thedisplay 412 of the head wearable device 140 is shown. In response to theuser selecting the user-selectable option to send a “UAS”, the processor402 may update the stream of image data to include graphical content ofuser-selectable “UAS STATUS” information 1402. The UAS Statusinformation 1402 may include information related to UASs 142-1 and UAS142-2, which are in the vicinity of the unknown threat 1104. Forexample, the UAS Status information 1402 may include informationassociated with each UAS's location, estimated time to arrive at theunknown threat 1104, a reconnaissance payload (e.g., types of opticalsensors and surveillance sensors onboard), and an attack payload (e.g.,types and quantities of weapons onboard).

Referring now to FIG. 15, a view of an exemplary image displayed by thedisplay 412 of the head wearable device 140 is shown. By looking nearthe user-selectable option of a particular UAS of the UAS Statusinformation 1402 and/or by speaking a voice command, the user inputsystem (e.g., the eye tracking system 408 and/or the voice recognitionsystem 418) may output user selection data to the processor 402indicative of the user selecting a particular “UAS” to deploy. Inresponse, the processor 402 may update the stream of image data toinclude graphical content of a “DEPLOY UAS” confirmation 1502 and outputthe updated stream of image data to the display 412. Additionally, theprocessor 402 may output a command to deploy the particular UAS.

Referring now to FIG. 16, a view of an exemplary image displayed by thedisplay 412 of the head wearable device 140 is shown. Upon the deployedUAS surveilling the unknown threat 1104, the deployed UAS may transmitsensor feed data (e.g., from optical sensors and/or surveillancesensors), which may be routed to the head wearable device 140. Uponreceiving the sensor feed data from the deployed UAS, the processor 402may update the stream of image data to include sensor feed graphicalcontent 1602 associated with the sensor feed data, and such graphicalcontent may be depicted alongside the unknown threat 1104. For example,the sensor feed graphical content 1602 may include images of the unknownthreat 1104.

Referring now to FIG. 17, a view of an exemplary image displayed by thedisplay 412 of the head wearable device 140 is shown. By looking nearthe user-selectable depiction of the unknown threat 1104 and/or byspeaking a voice command, the user input system (e.g., the eye trackingsystem 408 and/or the voice recognition system 418) may output usercommand data to the processor 402 indicative of the user commanding thatthe unknown threat 1104 be classified as an enemy. In response, theprocessor 402 may update the stream of image data to include graphicalcontent of a “CLASSIFY ENEMY” confirmation 1702 and output the updatedstream of image data to the display 412. Additionally, the processor 402may route the command to classify as an enemy to another computingdevice (e.g., 112) onboard the aircraft 102 or a device off of theaircraft 102.

Referring now to FIG. 18, a view of an exemplary image displayed by thedisplay 412 of the head wearable device 140 is shown. In response to theclassify as an enemy command, the processor 402 may update the stream ofimage data, such that the user-selectable depiction of the unknownthreat 1104 is updated to be indicative of an enemy classification, andoutput the updated stream of image data to the display 412. For example,by classifying the user-selectable depiction of the unknown threat 1104as an enemy, the color of the user-selectable depiction of the unknownthreat 1104 may change to red (for enemy) from yellow (for unknown),whereas teamed or friendly assets may be depicted as green.

While FIGS. 11-18 depict views of various exemplary images displayed bythe display 412 of the head wearable device 140 for one possiblesequence of actions, those of ordinary skill in the art will appreciatethat the head wearable device 140 according to the inventive conceptsdisclosed herein may be used by an operator for any of variousscenarios, such as for facilitating other MUM-T operations.

For example, in one exemplary scenario of facilitating MUM-T operations,a manned operator of the aircraft 102 may be responsible for pilotingthe aircraft 102 and for controlling four unmanned teamed assets (e.g.,four UASs 142). The aircraft 102 may be flying in a congestedbattlespace in a formation such that the four UASs 142 are flying aheadof the manned aircraft 102. The operator may be wearing the headwearable device 140, which, for example, may be implemented as anaugmented reality head wearable device with a see-through display. Asthe operator is flying, the operator can use the augmented reality headwearable device's overlaid graphical symbology to keep the operator'seyes focused out of the cockpit while being able to quickly locate teammembers, as well as other friendly, enemy, unknown, and neutral groundand air assets. As the operator scans the airspace, virtual symbols aredisplayed by the head wearable device 140 360° about the aircraft,overlaid on top of real aircraft locations. This allows the operator tovisualize the location of battlespace assets, even if the assets arebeyond the operator's natural line of sight. If the operator looks atone of the teamed assets (e.g., UASs 142), additional symbology appearsproviding information about the assets current operational and sensorsystem status. Additionally, the head wearable device 140 may displays ayellow dome on the ground, indicating an unknown ground threat. Theoperator may look at the displayed yellow dome to select the virtualsymbol associated with the unknown threat. When the yellow dome isselected, the head wearable device 140 may display a menu indicatingavailable team assets (e.g., UASs 142) that the operator can deploy tocapture intelligence on the unknown threat. The operator can select(e.g., via the eye tracking system 408 and/or the voice recognitionsystem 418) an item from the menu. For example, the operator may select“Deploy UAS 1 FLIR able” to dispatch one of the UASs 142 configured withFLIR cameras. Once the operator dispatches a teamed asset, a command issent to the teamed asset so that the teamed asset automatically performsthe desired action. Once the deployed asset is close enough to collectintelligence on the unknown threat, the sensor information may betransmitted to the aircraft 102 manned by the operator, and a video feedof the sensor information may be displayed by the head wearable device140 in a small window near (e.g., adjacent and/or above) the yellow domeindicating the unknown threat. Once the user views the video of thesensor information, the operator can use a voice command (e.g., via thevoice recognition system 418) to classify the unknown threat asfriendly, enemy, neutral. For example, the voice command may be“Classify Enemy” to classify the unknown threat as an enemy.

Referring now to FIG. 19, an exemplary embodiment of a method 1900according to the inventive concepts disclosed herein may include one ormore of the following steps. Additionally, for example, some embodimentsmay include performing one more instances of the method 1900iteratively, concurrently, and/or sequentially. Additionally, forexample, at least some of the steps of the method 1900 may be performedin parallel and/or concurrently. Additionally, in some embodiments, atleast some of the steps of the method 1900 may be performednon-sequentially.

A step 1902 may include outputting, by at least one processor of a headwearable device, a stream of image data to a display of the headwearable device for presentation to a user of the head wearable device,the image data associated with images aligned with a determined positionand a determined orientation of a head of the user relative to anenvironment, the images including at least one user-selectable depictionof at least one unmanned vehicle.

A step 1904 may include receiving, by the at least one processor of thehead wearable device, user input data from a user input system, whereinthe user input data includes user selection data associated with aselected unmanned vehicle of the at least one unmanned vehicle.

A step 1906 may include updating, by the at least one processor of thehead wearable device, the stream of image data associated with theimages such that the images further include a depiction of informationassociated with the selected unmanned vehicle.

Further, the method may include any of the operations disclosedthroughout.

As will be appreciated from the above, embodiments of the inventiveconcepts disclosed herein may be directed to a method, a system, and atleast one head wearable device.

As used throughout and as would be appreciated by those skilled in theart, “at least one non-transitory computer-readable medium” may refer toas at least one non-transitory computer-readable medium (e.g., memory110, memory 116, memory 212, memory 404, memory 506, memory 606, memory706, memory 116-1, storage 118, storage 214, storage 406, storage 508,storage 608, storage 708, or a combination thereof; e.g., at least onecomputer-readable medium implemented as hardware; e.g., at least onenon-transitory processor-readable medium, at least one memory (e.g., atleast one nonvolatile memory, at least one volatile memory, or acombination thereof; e.g., at least one random-access memory, at leastone flash memory, at least one read-only memory (ROM) (e.g., at leastone electrically erasable programmable read-only memory (EEPROM)), atleast one on-processor memory (e.g., at least one on-processor cache, atleast one on-processor buffer, at least one on-processor flash memory,at least one on-processor EEPROM, or a combination thereof), or acombination thereof), at least one storage device (e.g., at least onehard-disk drive, at least one tape drive, at least one solid-statedrive, at least one flash drive, at least one readable and/or writabledisk of at least one optical drive configured to read from and/or writeto the at least one readable and/or writable disk, or a combinationthereof), or a combination thereof).

As used throughout, “at least one” means one or a plurality of; forexample, “at least one” may comprise one, two, three, . . . , onehundred, or more. Similarly, as used throughout, “one or more” means oneor a plurality of; for example, “one or more” may comprise one, two,three, . . . , one hundred, or more. Further, as used throughout, “zeroor more” means zero, one, or a plurality of; for example, “zero or more”may comprise zero, one, two, three, . . . , one hundred, or more.

In the present disclosure, the methods, operations, and/or functionalitydisclosed may be implemented as sets of instructions or softwarereadable by a device. Further, it is understood that the specific orderor hierarchy of steps in the methods, operations, and/or functionalitydisclosed are examples of exemplary approaches. Based upon designpreferences, it is understood that the specific order or hierarchy ofsteps in the methods, operations, and/or functionality can be rearrangedwhile remaining within the scope of the inventive concepts disclosedherein. The accompanying claims may present elements of the varioussteps in a sample order, and are not necessarily meant to be limited tothe specific order or hierarchy presented.

It is to be understood that embodiments of the methods according to theinventive concepts disclosed herein may include one or more of the stepsdescribed herein. Further, such steps may be carried out in any desiredorder and two or more of the steps may be carried out simultaneouslywith one another. Two or more of the steps disclosed herein may becombined in a single step, and in some embodiments, one or more of thesteps may be carried out as two or more sub-steps. Further, other stepsor sub-steps may be carried in addition to, or as substitutes to one ormore of the steps disclosed herein.

From the above description, it is clear that the inventive conceptsdisclosed herein are well adapted to carry out the objects and to attainthe advantages mentioned herein as well as those inherent in theinventive concepts disclosed herein. While presently preferredembodiments of the inventive concepts disclosed herein have beendescribed for purposes of this disclosure, it will be understood thatnumerous changes may be made which will readily suggest themselves tothose skilled in the art and which are accomplished within the broadscope and coverage of the inventive concepts disclosed and claimedherein.

1. A head wearable device, comprising: a display implemented in or onthe head wearable device, the display configured to present imagery to auser of the head wearable device; a head tracking system implemented inor on the head wearable device, the head tracking system configured to:determine a position and an orientation of a head of the user of thehead wearable device relative to an environment; and output position andorientation data associated with the determined position and thedetermined orientation of the head of the user relative to theenvironment; a user input system comprising at least one of an eyetracking system or a voice recognition system, the user input systemconfigured to: detect user inputs of the user of the head wearabledevice; and output user input data; and at least one processorcommunicatively coupled to the head tracking system, the user inputsystem, and the display, the at least one processor configured to:output a stream of image data to the display for presentation to theuser, the image data associated with images aligned with the determinedposition and the determined orientation of the head of the user relativeto the environment, the images including at least one user-selectabledepiction of at least one teamed asset; receive the user input data fromthe user input system, wherein the user input data includes userselection data associated with a selected teamed asset of the at leastone teamed asset; and update the stream of image data associated withthe images such that the images further include a depiction ofinformation associated with the selected teamed asset.
 2. The headwearable device of claim 1, wherein the depiction of informationassociated with the selected teamed asset includes a depiction of sensorfeed data captured by at least one sensor of the selected teamed asset.3. The head wearable device of claim 1, wherein the informationassociated with the selected teamed asset includes information of atleast one of status, tasks, or capabilities of the selected teamedasset.
 4. The head wearable device of claim 1, wherein the user inputdata further includes user command data associated with a command tocontrol or manage the selected teamed asset, wherein the at least oneprocessor is further configured to: output a command to control ormanage the selected teamed asset.
 5. The head wearable device of claim1, wherein the image data is battlespace image data associated withimages of a battlespace.
 6. The head wearable device of claim 5, whereinthe images associated with the image data further include at least onedepiction of at least one battlespace feature.
 7. The head wearabledevice of claim 5, wherein the images associated with the image datafurther include at least one user-selectable depiction of at least onebattlespace object, wherein the user input data further includes userselection data associated with a selected battlespace object of the atleast one battlespace object, wherein the at least one processor isfurther configured to: update the stream of image data associated withthe images such that the images further include a depiction ofinformation associated with the selected battlespace object.
 8. The headwearable device of claim 7, wherein the user input data further includesuser command data associated with a command to surveil the selectedbattlespace object, wherein the at least one processor is furtherconfigured to: output a command to control or manage a particular teamedasset of the at least one teamed asset to surveil the selectedbattlespace object.
 9. The head wearable device of claim 7, wherein theuser input data further includes user command data associated with acommand to attack the selected battlespace object, wherein the at leastone processor is further configured to: output a command to control aparticular teamed asset of the at least one teamed asset to attack theselected battlespace object.
 10. The head wearable device of claim 7,wherein each of the at least one user-selectable depiction of the atleast one battlespace object includes graphical information indicativeof whether each of the at least one battlespace object is classified asfriendly, neutral, enemy, or unknown threat.
 11. The head wearabledevice of claim 10, wherein the selected battlespace object is currentlyclassified as unknown threat, wherein the user input data furtherincludes user command data associated with a command to classify theselected battlespace object, wherein the at least one processor isfurther configured to: output a command to classify the selectedbattlespace object as friendly, neutral, or enemy.
 12. The head wearabledevice of claim 1, wherein the display is an augmented reality display.13. The head wearable device of claim 12, wherein the display is linkedto a head down display configured to simultaneously present, to theuser, information synched with the head wearable device.
 14. The headwearable device of claim 1, wherein the display is a virtual realitydisplay.
 15. The head wearable device of claim 1, wherein the headwearable device is implemented onboard an aircraft, wherein the headwearable device is worn by the user onboard the aircraft.
 16. The headwearable device of claim 1, wherein the head wearable device isimplemented at a control station, wherein the head wearable device isworn by the user at the control station.
 17. The head wearable device ofclaim 1, wherein the images include a depiction of informationclassified to a predetermined classification level, wherein the headwearable device is implemented in an unclassified environment, whereinthe images are only viewable by the user of the head wearable device.18. The head wearable device of claim 1, further comprising a biometricsensor configured to sense biometric information associated with theuser, wherein the at least one processor is further configured to verifyan identity of the user of the head wearable device based at least onthe biometric information and determine a predetermined classificationlevel of the user based at least on the identity, wherein the stream ofimage data output to the display corresponds to the predeterminedclassification level of the user.
 19. A method, comprising: outputting,by at least one processor of a head wearable device, a stream of imagedata to a display of the head wearable device for presentation to a userof the head wearable device, the image data associated with imagesaligned with a determined position and a determined orientation of ahead of the user relative to an environment, the images including atleast one user-selectable depiction of at least one teamed asset;receiving, by the at least one processor of the head wearable device,user input data from a user input system, wherein the user input dataincludes user selection data associated with a selected teamed asset ofthe at least one teamed asset; and updating, by the at least oneprocessor of the head wearable device, the stream of image dataassociated with the images such that the images further include adepiction of information associated with the selected teamed asset. 20.A system, comprising: a computing device comprising a processor, theprocessor configured to output data; and a head wearable devicecommunicatively coupled to the computing device, the head wearabledevice comprising: a display implemented in or on the head wearabledevice, the display configured to present imagery to a user of the headwearable device; a head tracking system implemented in or on the headwearable device, the head tracking system configured to: determine aposition and an orientation of a head of the user of the head wearabledevice relative to an environment; and output position and orientationdata associated with the determined position and the determinedorientation of the head of the user relative to the environment; a userinput system comprising at least one of an eye tracking system or avoice recognition system, the user input system configured to: detectuser inputs of the user of the head wearable device; and output userinput data; at least one processor communicatively coupled to the headtracking system, the user input system, and the display, the at leastone processor configured to: receive the data from the processor of thecomputing device; based at least in part on the received data, output astream of image data to the display for presentation to the user, theimage data associated with images aligned with the determined positionand the determined orientation of the head of the user relative to theenvironment, the images including at least one user-selectable depictionof at least one teamed asset; receive the user input data from the userinput system, wherein the user input data includes user selection dataassociated with a selected teamed asset of the at least one teamedasset; and update the stream of image data associated with the imagessuch that the images further include a depiction of informationassociated with the selected teamed asset.